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Applied Research Consultants (ARC) is a graduate student run consulting firm that provides 
experience to students in evaluation and consultation. An overview of this program has been 
compiled in order to serve as a model of a graduate training practicum that could be applied to 
similar programs or aid in the development of such programs. Key performance aspects are 
described in detail to assist in implementation by departments in various higher education programs. 


Teaching Evaluation: 

A Student Run Consulting Firm 

A consulting practicum for graduate students is 
rare in advanced education, but it is highly needed in 
order to develop students into professionals in the field 
of evaluation (Belli, 2001; Morris, 1992; Trevisan, 
2002). According to Shadish, Cook, and Leviton, we 
can evaluate the effectiveness and efficiency of 
anything, “including evaluation itself’ (1991, p. 19). 
Graduate students need the opportunity to participate in 
evaluative work to be able to gain valuable knowledge 
and real-life experience. Cole (1995) claims there is a 
need for improvement in instructional teaching with 
respect to research methodology, and this could be 
improved by involving students in a practicum that 
gives them the hands on experience they need. 

There are similarities and differences between 
research and evaluation. Russ-Eft and Preskill (2001) 
identified commonalities in multiple definitions of 
evaluation. Evaluation is a systematic planned activity 
involving the collection of data regarding societal 
issues in general and, more specifically, individual 
programs for the purpose of direct use in enhancing 
knowledge and decision making, as opposed to general 
research that examines questions that are not 
necessarily applied to programs. Russ-Eft and Preskill 
list three core differences between research and 
evaluation: evaluation is 1) initiated for a different 
purpose, 2) conducted for a different audience and 
addresses clients’ needs, and 3) communicated in a 
different way. 

A strong training in systematic research methods, 
data management, and statistical techniques provides a 
skill base for conducting evaluation. However, research 
methods alone do not constitute the total skills needed 
to conduct evaluations. Training in evaluation requires 
a well-built base of research methodology combined 
with an understanding of the differences between 
research and evaluation. Practical experience is needed 
to master tailoring research to answer client based 
questions, to understand the reality of program 


evaluation feasibility, and to communicate results to 
decision makers. 

Experiential practicum in evaluation is necessary, 
and universities need to seek out additional ways to 
educate students in research and consulting (Chelimsky, 
1997; Dallimore & Souza, 2002). Consulting is the 
communication process an evaluator takes with his or 
her clients, such as meeting with organizational leaders 
to get their interpretation of the program and its 
problems, and dissemination of evaluative information. 
Evaluators tend to adopt different forms of consulting 
practices based on their theoretical background and 
experiences (Shadish et al., 1991). 

This paper reviews a student run consulting firm, 
Applied Research Consultants (ARC), in order to 
demonstrate how to incorporate an evaluation 
practicum in a higher education setting. 

ARC directly involves students in the process of 
evaluation in a mentor supported environment. 
Evaluations from beginning to end are conducted by 
students, but all projects, from simple data collection to 
multi-year assignments, are subjected to quality control 
through departmental faculty as well as advanced 
students in the program. The main goal behind the 
program is providing experience to Master’s and Ph.D. 
students in planning, managing, and conducting applied 
research and evaluation while using consulting 
practices. The aim of this paper, therefore, is to provide 
instructors and university department heads with a basic 
structure that could be utilized in order to implement 
similar practica. 

Evident Practices 

Dallimore and Souza (2002) presented a class 
design model that looks at teaching evaluation and 
consulting to graduate students with the emphasis on 
enhancing students’ experiential learning. However, 
this design was made for a single-semester course, 
which is typically seen in student evaluation training. 
Typically such “advanced training” rarely lasts beyond 
the semester, and students in these programs generally 
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do not work from beginning to end on a project (Belli, 
2001; Trevisan, 2004). Therefore, the time spent on 
evaluation training needs to be increased in order to 
encompass the skills necessary to be successful in this 
type of work. 

Training graduate students with consulting 
experience tends to focus on conducting and 
interpreting statistical analyses or planning research 
designs. These training assignments generally do not 
provide instruction on measurement theories, 
construction of measures, or presentations at workshops 
(Belli, 2001). Further, previous research has pointed out 
areas where current graduate student evaluation 
practicum is weak (for a review of programs see 
Trevisan, 2004). While authors (Belli, 2001; 
Chelimsky, 1997; Morris, 2002; Preskill, 2000) have 
stressed the importance of practical experiences in 
training evaluation, Trevisan (2004) found only a few 
programs offering practical experience beyond single 
semester class projects. 

The key components in teaching evaluation are 
theory, methodology, conceptualization, and practical 
experience (Levin-Rozalis & Rosenstein, 2003), and 
ARC facilitates the learning of all four of these 
components while also providing the additional 
practical experience recommended in the literature for 
advanced student training. For example, on any one 
project students consult with clients; develop a contract 
and budget; develop methodologies and measures; 
implement the study; gather, analyze, and interpret 
data; and present the information in written and oral 
reports. Previous research has pointed to the necessity 
of keeping the basic research design in mind when 
training graduate students in evaluation (Dallimore & 
Souza, 2002; Preskill, 2000). The typical progression in 
research methodology begins with problem 
identification through data collection, analysis, 
interpretation, and translation. The ARC program 
generally uses this type of progression while 
implementing research, but research design is not 
always necessary. For example, if a client seeks 
assistance after the data have already been collected, 
which frequently occurs (Belli, 2001; Nadler & 
Cundiff, 2000), the study is past the need for 
methodological design. To fully understand ARC and 
its effectiveness as an educational tool with the goal of 
increasing the knowledge base in graduate student 
practica, it is important to define the ARC practicum 
and represent how it works. 

Structure of ARC 

ARC is a graduate student run consulting firm, 
located in Southern Illinois University’s Psychology 
Department. ARC’S goals are supported by practical 
experience with clients and report writing, the most 


often mentioned “gaps” between expectations in 
evaluative occupations and applicant skills when 
recruiters are hiring new evaluators (Dewey, 
Montrosse, Schroeter, Sullins, & Mattox, 2006). ARC’S 
students are composed of graduate students in Applied 
Psychology; however, the program’s structure could be 
extended to a variety of educational programs such as 
management, workforce education, or educational 
psychology. In fact, any program interested in 
investigating the effectiveness and efficacy of different 
programs, services, products, or organizations could use 
ARC as a model to develop an intensive program that 
would enhance students’ skills and development in 
research and evaluation. 

ARC provides many services to its customers such 
as measurement development, data collection and 
analysis, report writing, interviewing, and focus group 
assessments. These processes are all a part of program 
evaluation (Shadish et ah, 1991). However, the 
evaluation’s focus differs drastically depending on the 
needs of the clients. For instance, ARC has served as 
evaluator on a large grant in which students developed 
a needs assessment, created a logic model depicting the 
program, developed trainings, and created a research 
agenda that offered informative outcomes of the 
program. ARC has also served as a guide for a private 
company to become a research institution by providing 
key information about research intuitions and 
developing models to depict the strengths and 
weaknesses of the organization. These are just two 
examples of the types of work ARC has conducted in 
recent years, and there are many types of jobs and 
clients that are included in ARC’S repertoire. Programs 
that are developed based on a model such as ARC 
should be encouraged to seek out clients both from 
inside and outside of the academic community. 

What makes ARC’S design different from other 
programs is that the students participate in the course 
for over two full years, which enhances experience by 
allowing them to follow projects extending across 
multiple semesters. This process is important in 
evaluation training, since “real world” consulting jobs 
tend to take anywhere from three months to many 
years. Students who are training for such work need to 
be able to work with clients and on a project for as long 
as the job entails. 

ARC’S Class 

While the work in ARC is conducted by graduate 
students, they are supervised by a director who is 
traditionally a tenured professor at the university with 
other faculty serving as resources on an as-needed 
basis. The daily lives of the ARC associates are 
fragmented and have few stable patterns of work or 
activities. The “class” of ARC, led by the director, 
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meets for two hours weekly to keep everyone in the 
program up-to-date with projects and business related 
matters. Discussions range from simple administrative 
matters to thorough and complex think-tank sessions on 
methodology, theory, and client relations. 

Students learn through assigned readings that cover 
evaluation concepts and through experiencing problems 
that come up while working with clients, such as 
communication issues. 

Solving problems in a group meeting is a rare 
procedure of evaluation training. Most evaluation 
practica tend to solve problems by one-on-one meetings 
with faculty (Belli, 2001). However, presenting 
problems in a group setting allows for students to learn 
about the issues other members are facing and for 
students to give each other advice. 

ARC Projects 

Projects are also disseminated at class meetings 
where contact with prospective clients is arranged and 
given to a senior associate charged with leading the 
project. Once projects are assigned to an associate, the 
student assumes responsibility for project completion. 
The choice of leader for a project depends of the type of 
project and student availability. Typically projects are 
matched with students who have the skills and the 
experience to assess and manage the situation. 
Consideration of students’ substantive interests is also 
assessed when assigning projects; while student input is 
an important factor in project leader choices the final 
decision lies with the faculty director. 

The project leader has a team of associates who 
either volunteer to be on the project or are assigned to 
the project by the director. Volunteering usually occurs 
when students have interest in the project’s area or they 
are interested in the methodological procedures that are 
being used for that project. For example, a student may 
be interested in working on an evaluation being 
conducted for a local hospital due to interest in the 
health field. Likewise, some students would volunteer 
for a project involving focus groups if they were 
interested in gaining experience in qualitative data 
collection procedures. While students are granted 
latitude in determining what projects they are on, the 
faculty director monitors the process to ensure each 
project is adequately staffed and each student gains 
experiences as needed. 

The project leader then instructs his or her team on 
how to administer the evaluation by passing out 
assignments to the associates on the team and 
coordinating their efforts. The evaluation may involve 
developing a survey that would be used to conduct 
research on the evaluand or the object being evaluated. 
For instance, various measures may be devised to 
ascertain the effectiveness of a new program that is 


directed at reducing childhood obesity. The leader may 
start construction of the survey by consulting with a 
client (usually directors of the programs or 
organizations), then team members would meet 
together in order to ensure the quality and thoroughness 
of the measure being developed. The team would also 
assist the project leader in constructing the survey to 
formats acceptable for mailing and/or on-line 
distribution. Projects go through different stages for 
each evaluation, since every evaluation is unique. 

Tri-level Involvement 

ARC is constructed based on three levels of student 
involvement. The first level is a shadowing period 
which allows students to directly observe the practices 
of the program without having the responsibility of 
working on projects. Experiential training often stops at 
this level of involvement in evaluation programs, which 
includes role playing, observing faculty conducting 
evaluations, and participating in class projects 
(Trevisan, 2004). ARC’S second level of student 
involvement has students working as associates on 
projects and receiving guidance and instruction from 
senior associates in the program. Assistance from senior 
members is considered to be a vertical practicum, with 
advanced students training novice students. Intuitional 
knowledge is retained by processes and lessons learned 
that are passed down through the vertical practicum. 
The vertical practicum enables learning to be facilitated 
in the program and insures that time is not lost due to 
the relearning of simpler processes such as on-line 
survey development and database management. The 
vertical practicum promotes teaching students about 
leadership and organization; for more information on 
this subject see Nadler and Cundiff (2009). 

The third and final level of ARC is where the 
program extends beyond a traditional training 
practicum in evaluation. Students, having completed the 
first two levels, then work as senior associates who are 
in charge of entire evaluation and consulting 
assignments as project leaders. At this level, student 
involvement includes all aspects of starting up, working 
on, and completing a consulting project. This can 
include, but is not limited to: contacting and meeting 
with clients, writing budgets and contracts, devising 
appropriate methodologies, collecting data, analyzing 
data, and writing and distributing reports. Once again, 
all of these activities are conducted within a safety net 
offered by faculty oversight. 

The Students 

There are usually 10-12 student associates in ARC 
per semester. These students are charged with running 
the organization. The position as an associate is usually 
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the first professional working experience for many of 
these students. For instance, some come into the program 
right out of their undergraduate careers, and they have 
reported being a little awed and fearful of the tasks 
before them. These students begin at the first level during 
the second semester of their Ph.D. program by 
shadowing projects. Since new students generally report 
anxiety about the level of responsibility and commitment 
that the program requires of them, it is important 
therefore to introduce students to such a practician with 
care. 

Other students have had some previous experience 
working in consulting or have advanced degrees which 
assist them in the process of entering into the program. 
The applied graduate program, of which ARC is an 
active part, balances each year’s cohort with students 
straight out of undergraduate school with students having 
research or practical experience in evaluation. Both 
groups of students taken together create a dynamic 
evaluation group that is grounded in an applied research 
culture. 

The Place 

The ARC lab is a designated area where associates 
have 24 hour access to learn, teach, and conduct 
evaluations. The lab consists of a computer room, a 
conference room, a library, and a break room. There is a 
server dedicated to up-to-date technological programs 
that assist and protect the evaluators’ work. Tasks such 
as compiling data, analyzing research, and gathering 
information across the country should be easily 
conducted with the resources located in these types of 
labs. 

Supplies in the lab are readily available to the 
students and have been acquired over the 25 years that 
ARC has been running. Time in these types of programs 
is necessary to accumulate resources. Most resources are 
within reasonable price ranges, so start up costs should 
not be feared. Typically, what is needed to begin is space 
to work and hold meetings, computer and printer access, 
an internet server, statistical software, and guidance from 
faculty. Most of these resources will already be available 
in existing faculty graduate or undergraduate research 
labs. However, it is important to give such a program the 
feeling of being an independent firm, therefore setting 
dedicated space for training should be a priority. 
Additionally, research on consulting practica has found 
that usually programs such as these should have one to 
three invested faculty members to assist students with the 
practicum experience (Belli, 2001). 

Generating Resources 

Resources are an immediate concern for faculty 
and administrators whom are considering 


implementation of a practicum that requires a full scale 
lab. However, such programs can be self-sustaining by 
primarily focusing on providing training for the students 
and not on income generation (Belli, 2001). In ARC, for 
instance, the income generated from the associates’ work 
can all be invested back into the program. ARC charges 
$40.00 per student work hour, with typical total project 
fees ranging from $500 to $5,000. ARC’S fees are at the 
lower extreme of evaluation rates (Jarosewich, 2006), 
providing clients with an alternative to costly evaluation 
firms. ARC works internally within the university and 
with local businesses, and it usually brings in between 
$35,000 to $55,000 a year. 

Generally the students involved in ARC decide, by a 
democratic voting process, how the income of the firm is 
spent. However, most of the income goes towards 
keeping the basic necessities of the organization running, 
such as having office and technology support and buying 
equipment. The profits from the business then go 
towards sending the students to an annual evaluation 
conference held by the American Evaluation Association 
(AEA), which is thought of as one of the more important 
aspects of the program by the students. 

An internal grant funded by ARC is also available to 
students, whereby they are able to receive funding for 
professional development. This is another aspect of ARC 
that is unique when compared to other training practica: 
students actually work to bring money into the program 
and profits can directly benefit their personal research. 

Process 

ARC associates handle a variety of different types of 
research and evaluation procedures. In other words, the 
program specializes in survey design (paper and on-line) 
and dissemination, market research, program evaluation 
(general), focus group facilitation, in-depth interviews 
(cognitive and semi-structured), needs assessments, job 
analyses, performance appraisals, and personnel training, 
to name a few. However, job type (consultation and/or 
evaluation) typically depends on what clients need; 
therefore, students utilize their education from classes 
and prior ARC experience to facilitate quality work at 
half the price of competitors. The use of classroom 
knowledge in evaluation has its benefits, and the practice 
students gain from the consulting and evaluation tend to 
“reinforce theoretical material taught in core research 
course(s)” (Cole, 1995, p. 159). 

Conclusion 

Through a consulting/training practicum, ARC has 
the structure to support itself while providing 
appropriate evaluation experience to students. The way 
the program functions is fitting for academic 
institutions interested in educating graduate students, 
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whether at the master’s or doctoral level, in consulting 
and evaluation. This review was intended to serve as a 
guide to fellow educators interested in effectively 
training graduate students in the evaluation field. Some 
issues have been documented on negative reactions 
towards such programs within university settings, such 
as departments receiving lower financial support due to 
less reported research hours from faculty who are 
devoted to directing the programs or issues involving 
ethical review boards (Beck & Kosnik, 2002). These 
concerns do not outweigh the positive effects of such 
programs though, exemplified in the amount of 
experience gained by the students and faculty involved. 
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